Requirements Engineering: Foundation for Software Quality by Nazim Madhavji & Liliana Pasquale

Author:Nazim Madhavji & Liliana Pasquale
Language: eng
Format: epub
ISBN: 9783030444297
Publisher: Springer International Publishing

7 Conclusion and Future Work

We have presented Kyōryoku, a crowdsourcing method for identifying and classifying user requirements – more precisely, requirements-relevant information – in online user feedback through crowd work. Kyōryoku was tested on 1,000 app store reviews, which were analyzed and classified by over 400 crowd workers.

Based on the outcomes of Phases 1 & 2 of Kyōryoku, we can confidently state that crowd workers are able to distinguish between useful and useless reviews (H1). The crowd workers achieved precision rates of 93% and 88% and recall rates of 84% and 81%, respectively, in these phases. Although there is no automated technique that serves as a baseline, further research is needed to compare against algorithms based on automated spam detection in app reviews [2].

When we consider the ability of the crowd workers to correctly assign user reviews to different requirement categories (H2), the results are positive, but inevitably not as good as the binary useful/useless classification. The overall accuracy was 74% for the five categories that we deemed suitable for crowdsourced classification: “Feature”, “Stability”, “Performance”, (other) “Quality”, and “None”. Interestingly, for the 85 fragments with perfect agreement among all six taggers, we could observe 100% accuracy. We have not tested Kyōryoku against automated classifiers yet. These results seem to be at least as good as optimized automated classifiers of NFRs [21], which achieve an accuracy of 70%.

H3 concerned the feasibility and cost-effectiveness of Kyōryoku to extract RE-relevant contents from online user feedback. We were able to show the feasibility of such a method through the tasks we composed for the crowd workers to carry out. In terms of cost-effectiveness, 1,000 reviews were fully processed through crowdsourcing for approximately $350 and in 5.4 h for all phases and sessions combined. On the other hand, creating a gold standard, i.e., tagging the data without crowd workers, required circa 20–30 person-hours. Although we cannot provide a conclusive answer to H3, the results suggest that Kyōryoku might be suitable for companies who wish to analyze user reviews about their products, but who do not have sufficient resources to hire an expert assessor.

This work presents a novel method for engaging a crowd to elicit user requirements from online user feedback, and paves the way for future work in this direction. Kyōryoku, which includes openly available task descriptions [33], can be taken as is and used by organizations who would like to classify a reviews dataset. Kyōryoku can be improved by changing the wording of the job description, the examples, and the classification taxonomy. To do so, it is imperative to complement our quantitative results with a qualitative analysis that reveals which utterances are most likely to lead to false positives and false negatives. We hope that future studies will take Kyōryoku as a baseline to improve upon; researchers can directly compare their automated or human-driven method using the gold standard we make available. Alternatively, it is possible to use this gold standard to train approaches based on ML. Also, it would be interesting to investigate whether the crowd can effectively use fine-grained taxonomies of quality requirements.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.